{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic training and testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "using MLJ\n",
    "using DataFrames\n",
    "\n",
    "task = load_boston()\n",
    "X, y = X_and_y(task);\n",
    "\n",
    "X = DataFrame(X) # or any other tabular format supported by Table.jl \n",
    "\n",
    "train, test = partition(eachindex(y), 0.7); # 70:30 split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A *model* is a container for hyperparameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# \u001b[0m\u001b[1mKNNRegressor{Float64} @ 1…90\u001b[22m: \n",
       "target_type             =>   Float64\n",
       "K                       =>   10\n",
       "metric                  =>   euclidean (generic function with 1 method)\n",
       "kernel                  =>   reciprocal (generic function with 1 method)\n",
       "\n"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_model=KNNRegressor(K=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wrapping the model in data creates a *machine* which will store training outcomes (called *fit-results*):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# \u001b[0m\u001b[1mMachine{KNNRegressor{Float64}} @ 9…72\u001b[22m: \n",
       "model                   =>   \u001b[0m\u001b[1mKNNRegressor{Float64} @ 1…90\u001b[22m\n",
       "fitresult               =>   (undefined)\n",
       "cache                   =>   (undefined)\n",
       "args                    =>   (omitted Tuple{DataFrame,Array{Float64,1}} of length 2)\n",
       "report                  =>   empty Dict{Symbol,Any}\n",
       "rows                    =>   (undefined)\n",
       "\n"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn = machine(knn_model, X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training on the training rows and evaluating on the test rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Info: Training \u001b[0m\u001b[1mMachine{KNNRegressor{Float64}} @ 9…72\u001b[22m.\n",
      "└ @ MLJ /Users/anthony/Dropbox/Julia7/MLJ/src/machines.jl:69\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "8.090639098853249"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fit!(knn, rows=train)\n",
    "yhat = predict(knn, X[test,:])\n",
    "rms(y[test], yhat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, in one line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8.090639098853249"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate!(knn, resampling=Holdout(fraction_train=0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Changing a hyperparameter and re-evaluating:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8.41003854724935"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_model.K = 20\n",
    "evaluate!(knn, resampling=Holdout(fraction_train=0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Systematic tuning as a model wrapper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple example of a composite model is a homogeneous ensemble. Here's a bagged ensemble model for 20 K-nearest neighbour regressors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# \u001b[0m\u001b[1mDeterministicEnsembleModel @ 5…24\u001b[22m: \n",
       "atom                    =>   \u001b[0m\u001b[1mKNNRegressor{Float64} @ 1…90\u001b[22m\n",
       "weights                 =>   0-element Array{Float64,1}\n",
       "bagging_fraction        =>   0.8\n",
       "rng_seed                =>   0\n",
       "n                       =>   20\n",
       "parallel                =>   true\n",
       "\n"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ensemble_model = EnsembleModel(atom=knn_model, n=20) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's simultaneously tune the ensemble's `bagging_fraction` and the K-nearest neighbour hyperparameter `K`. Since one of these models is a field of the other, we have nested hyperparameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Params(:atom => Params(:target_type => Float64, :K => 20, :metric => MLJ.KNN.euclidean, :kernel => MLJ.KNN.reciprocal), :weights => Float64[], :bagging_fraction => 0.8, :rng_seed => 0, :n => 20, :parallel => true)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params(ensemble_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To define a tuning grid, we construct ranges for the two parameters and collate these ranges following the same pattern above (omitting parameters that don't change):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Params(:atom => Params(:K => \u001b[0m\u001b[1mNumericRange @ 1…75\u001b[22m), :bagging_fraction => \u001b[0m\u001b[1mNumericRange @ 1…56\u001b[22m)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "B_range = range(ensemble_model, :bagging_fraction, lower= 0.5, upper=1.0, scale = :linear)\n",
    "K_range = range(knn_model, :K, lower=1, upper=100, scale=:log10)\n",
    "nested_ranges = Params(:atom => Params(:K => K_range), :bagging_fraction => B_range)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we choose a tuning strategy, and a resampling strategy (for estimating performance), and wrap these strategies around our ensemble model to obtain a new model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# \u001b[0m\u001b[1mDeterministicTunedModel @ 1…93\u001b[22m: \n",
       "model                   =>   \u001b[0m\u001b[1mDeterministicEnsembleModel @ 5…24\u001b[22m\n",
       "tuning                  =>   \u001b[0m\u001b[1mGrid @ 1…37\u001b[22m\n",
       "resampling              =>   \u001b[0m\u001b[1mCV @ 6…31\u001b[22m\n",
       "measure                 =>   nothing\n",
       "operation               =>   predict (generic function with 19 methods)\n",
       "nested_ranges           =>   Params(:atom => Params(:K => \u001b[0m\u001b[1mNumericRange @ 1…75\u001b[22m), :bagging_fraction => \u001b[0m\u001b[1mNumericRange @ 1…56\u001b[22m)\n",
       "report_measurements     =>   true\n",
       "\n"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tuning = Grid(resolution=12)\n",
    "resampling = CV(nfolds=6)\n",
    "\n",
    "tuned_ensemble_model = TunedModel(model=ensemble_model, \n",
    "    tuning=tuning, resampling=resampling, nested_ranges=nested_ranges)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fitting the corresponding machine tunes the underlying model (in this case an ensemble) and retrains on all supplied data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Info: Training \u001b[0m\u001b[1mMachine{MLJ.DeterministicTunedMo…} @ 1…05\u001b[22m.\n",
      "└ @ MLJ /Users/anthony/Dropbox/Julia7/MLJ/src/machines.jl:69\n",
      "\u001b[33mSearching a 132-point grid for best model: 100%[=========================] Time: 0:01:20\u001b[39m\n",
      "┌ Info: Training best model on all supplied data.\n",
      "└ @ MLJ /Users/anthony/Dropbox/Julia7/MLJ/src/tuning.jl:130\n"
     ]
    }
   ],
   "source": [
    "tuned_ensemble = machine(tuned_ensemble_model, X[train,:], y[train])\n",
    "fit!(tuned_ensemble);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{Symbol,Any} with 4 entries:\n",
       "  :measurements     => [7.03102, 6.09291, 6.05707, 5.93617, 5.86848, 5.73299, 5…\n",
       "  :models           => DeterministicEnsembleModel{Tuple{Array{Float64,2},Array{…\n",
       "  :best_model       => \u001b[0m\u001b[1mDeterministicEnsembleModel @ 3…49\u001b[22m\n",
       "  :best_measurement => 5.46102"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tuned_ensemble.report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best_model.bagging_fraction = 0.7272727272727273\n",
      "(best_model.atom).K = 100\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_model = tuned_ensemble.report[:best_model]\n",
    "@show best_model.bagging_fraction\n",
    "@show best_model.atom.K"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.0.2",
   "language": "julia",
   "name": "julia-1.0"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.0.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
